DOCUMENT RESUME 



ED 358 648 



EC 302 212 



AUTHOR 
TITLE 

INSTITUTION 

PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 



McLaughlin, John A. 

Special Education Program Evaluation: A Planning 
Guide, An Overview. CASE Commissioned Series* 
Council of Administrators of Special Education, Inc.; 
Indiana Univ., Bloomington. 
May 88 
HOp. 

CASE Research Committee, Indiana University, School 
of Education, Smith Research Center-IOOA, 2805 E. 
10th St., Bloomington, IN 47405 (Order No. PES-2, 
$15). 

Guides - Non~Cl ass room Use (055) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF01/PC05 Plus Postage. 

Data Collection; ^Disabilities; Elementary Secondary 
Education; '^Evaluation Methods; ^Evaluation 
Utilization; Information Sources; *Plcnning; Program 
Effectiveness; '^Program Evaluation; "Special 
Education 

^Evaluation Reports 



ABSTRACT 

This resource guide is intended to help in planning 
special education program evaluations. It focuses on: basic 
evaluation concepts, identification of special education decision 
makers and their information needs, specific evaluation questions, 
procedures for gathering relevant information, and evaluation of the 
evaluation process itself. Preliminary information discusses the 
nature of evaluation, the people involved, and ways to maximize the 
utilization of evaluation results. Then, the following eight steps to 
planning a local evaluation are detailed: (1) getting started; (2) 
describing the program; (3) writing evaluation questions; (4) 
planning collection of information; (5) planning analysis of 
evaluation data; (6) planning the evaluation report; (7) managing the 
evaluation; and (8) meta evaluation. Four appendices provide a meta 
evaluation checklist, a list of 8 references on evaluation 
utilization, a list of 11 specific strategies to enhance evaluation 
utilization, and 15 worksheets keyed to the 8 planning steps. (DB) 
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INTRODUCTION 



Purpose 

The purpose of this overview is to familiarize the reader with 
the basic principles and practices of program evaluation. The 
document primarily is intended for use as a resource or 
reference tool during a workshop on program evaluation. It 
is not expected that a reader will be able to simply read the 
document and conduct an evaluation. The focus of the document 
is on planning an evaluation. The overall goals are to enable 
the reader to: 



• understand evaluation concepts and what makes evaluations 
useful; 

• identify people who are going to make judgments about the 
special education program and their information needs; 

• determine evaluation questions which focus tha evaluation 
on particular aspects of the program; 

• plan for gathering information to enable judgments to be 
made about the program; and 

• evaluate the worth of the evaluation. 

What is Evaluation? 

Evaluation is a process through which evaluators gather 
information for decision makers. Therefore, the answer to the 
question "Why evaluate a program?" is clear. Evaluations are 
conducted because someone wants to know about the program. 

Generally evaluations are conducted to meet the needs of 
people who are going to be making judgments and/or decisions 
about the program. These people may be internal or external 
to the program. Often evaluations of programs are required 
by legislation as in the case of special education programs. 
Both state and federal legislation and attendant regulations 
call for the evaluation of special education programs. 
Required or not, the aims of evaluation are program 
improvement and sharing of success. Evaluation is a tool for 
advocacy. Through evaluation we ensure that appropriate 
programs are available to exceptional learners, their 
families , and those who serve them. 

Evaluation as a practice has been changing. In the beginning 
it was a goal oriented experimental enterprise. Now the 



expectations for useful evaluation include more formative 
aspects with an emphasis on changing programs based on 
feedback from the evaluation. 

Additionally the role and function of the evaluator have been 
modified. We used to think of the evaluator as being separate 
from the decision process. They were technicians who had the 
research expertise necessary to design information gathering 
systems, appropriate research designs and data analysis 
procedures. Program staff told the evaluator what information 
was needed and then the evaluator established the procedures 
required to obtain the information. The result was turned 
over to the client who would make the decision. 

New perspectives on evaluation call for the evaluator to be 
part of the decision team. In this sense the evaluation 
becomes a support system to the program. Evaluations and 
clients engage in an educative process to enable the 
evaluation to be productive. The client educates the 
evaluator about the purpose of evaluation, information needs 
and the context within which the evaluation will be conducted. 
The client makes the evaluator aware of political and 
programmatic constraints to the evaluation. On the other 
hand, the evaluator informs the client about valid, reliable 
and objective approaches to information collection. 
Evaluators assist the client in the interpretation of 
evaluation information. They make the client aware of the 
potential uses and misuses of the data. 

An important by-product of this new view of evaluation is the 
realization that the information generated from the evaluation 
is not the sole determinant of the decision mado by the 
client. Program staff combine this information with existing 
information from other sources including their perceptions of 
the real world; the political, social, economic and 
administrative factors associated with the program context. 

There are four facets to most evaluation efforts. The first 
is the statement of evaluation questions which guide the 
development of the evaluation plan. The questions signal the 
information that is needed by the decision maker. The next 
facet is information gathering. Once the questions are 
identified, the evaluation team designs and implements 
strategies to gather data to enable the questions to be 
answered. Data gathering processes may be formal or informal. 

The third facet of evaluation is the judgment. When data are 
gathered, they are put in a form that will allow people to 
interpret them in relation to some decision. Generally this 
interpretation is a comparative process that entails the 
comparison of the information to some standard. A standard 
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is an explicit or implicit statement of expectation or 
requirement. Standards come from many sources including: 
legislation, regulations, guidelines, court decisions, 
literature, and professional experience. Each person who is 
making a judgment about a program has some standard in mind. 
Often there are multiple judges in an evaluation and, at 
times, multiple standards which may be implicit or explicit. 1 

The final facet of the evaluation is the decision. 
Evaluations should lead to a decision about the program under 
consideration. People make decisions based on the outcome of 
the judgment process. The comparison between the data 
gathered (what is) and the standard (what should be) can lead 
to two possible conclusions: 

* the program looks like it should; or, 

* the program does not meet expectations. 

Given this information, five decisions may be made about the 
program: 

* continue the program until it meets the standard (we believe 
that our standard is viable or the standard is required; 
i.e. regulation) 

* keep the standard, but revise the program (try new 
materials, strategies, or train staff) 

* revise the standard (after looking at program 
implementation, we may believe the standard is unreasonable 
or unrealistic given the contextual factors of the program) 

* terminate the program (the discrepancy between what the 
program is and what we want it to be is so large that 
revisions would be too costly) 

* disseminate the program (if the program consistently meets 
the objectives or standards, it may be time to share it with 
others who are attempting to accomplish the same things). 

In summary: 

Evaluation Questions can be generated relative to program 
inputs, processes and/or outputs; 



A useful reference for identifying indicators of success in 
special education is: National RRC Panel on Effectiveness 
Indicators for Special Education. Effectiv eness Indicators 
for Special Education. A Reference Tool. Council of 
Administrators of Special Education, Indiana University, 
Bloomington, Indiana, 1986. 
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Information is collected to estimate actual performance and 
to determine unintended effects; 

Judgments are made about discrepancies between what should 
be and what is; and 

Decisions are made on the basis of the information* 

This overview should help the reader plan an evaluation that 
incorporates all four facets. 

People Involved in Evaluation. 

Evaluation is a people process. As can be seen from the above 
discussion, people ask questions about the program, gather 
data on the program, make judgments concerning the program, 
and, finally, make decisions about the program. Who are these 
people? 

Basically there are three groupe of people involved in the 
evaluation process. Eirst, there are the : .hpl. decision 
makers. These could be program administrators or the people 
who control the allocation of resources to the program. For 
special education this group might include: school district 
administrators, school board members, state department staff, 
representatives of funding agencies, and/or state and federal 
legislators. 

Other decision makers include program staff who make the 
day to-day decisions about the program. These persons take 
the resources allocated to the program and put them together 
to form a meaningful program. They make decisions about how 
to design and implement the program so that it meets the 
intents of the administrators and the needs of the client. 

The second group of people to be accounted for in an 
evaluation are the program influencors . This group includes 
those people who influence the administrators regarding the 
allocation of resources and influence the staff who are 
responsible for employing the resources in an appropriate way. 
For the local special education program these influencers 
might be students, parents, advocates, local advisory groups, 
community members, or politicians. 

The final group to be involved in the evaluation are the 
evaluators . These people are usually trained in the 
principles and practices of program evaluation. The 
evaluators should have a variety of skills including: program 
design, measurement, data analysis, report presentation, and 
group dynamics. 
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This last skill is perhaps the most important. Evaluation is 
a team sport. All the groups mentioned above should be 
involved in the evaluation. The roles played by the various 
actors should be determined on their skill and knowledge based 
on the needs of the evaluation process. There should be 
shared leadership and problem solving in an environment of 
mutual trust. All may be involved in any phase of the 
evaluation; including generation of evaluation needs and 
questions, data gathering, standards setting, formulation of 
evaluation reports, and, finally, determining how to use the 
results of the evaluation. 

The remainder of this paper will focus on key aspects of the 
evaluation process. The basic components of an evaluation 
plan will be discussed. The steps to conducting an evaluation 
of a local special education program will be presented. The 
concept of meta evaluation (evaluating the evaluation) is 
discussed as one of the critical steps. 
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MAXIMIZING THE UTILIZATION OF EVALUATION RESULTS 



Before progressing to a discussion of the form and stages of 
an evaluation of a local special education program it is 
necessary to identify factors which may affect the utilization 
of the evaluation results. Related discussion is found in the 
last section of this paper under meta-evaluation strategies. 2 

As noted throughout this paper it is difficult to think of a 
situation in which an evaluation is conducted without some 
purpose or use in mind. Several authors have defined use of 
evaluations as either instrumental or persuasive. 
Instrumental uses refer to immediate and observable uses of 
the results of the evaluation. The intervention strategies 
are changed or funding is modified. Persuasive uses are more 
difficult to discern. They relate to the decision maker's 
perception of the project being evaluated. Thus, while no 
immediate decisions can be observed, the evaluation serves to 
influence the perceptions of those interested in the program. 
Regardless of the kind of use intended by evaluators, it is 
possible to identify factors which appear to affect 
utilization. 

Factors W hich Affect Utilization 3 

According to Levi ton and Hughes (1981), there are five 
categories of factors which may affect the utilization of 
evaluation results: 

1. Relevance 

2 . Communi c a ti on 

3 . Information Processing 

4. Credibility 

5. Commitment to Advocacy 

The purpose of this section is to briefly discuss these 
factors presented by Levi ton and Hughes as they may be 
related to the utilization of evaluations directed at special 
education programs. It should be noted that these categories 
are interdependent and responses may address one or more 
factors. 

1. Relevance. 



z A partial list of references on utilization is set forth in 
Appendix B. 

3 A list of 'Strategies to Enhance Utilization 1 is found in 
Appendix C. 
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Many researchers studying utilization emphasize that the 
degree to which the evaluation accurately reflects the needs 
of potential users affects its use. To what extent are the 
needs of the clients embodied in the evaluation design? One 
program administrator was overheard saying that the design 
utilized was generic, one that could meet the needs of any 
program. As such, the anticipated results were so broad that 
they could not be used by staff to make decisions about 
improving the program. 

Timelines of the evaluation reports also affect relevance. 
If the report does not reach the user when a decision is about 
to be made, then it is unreasonable to expect that the 
decision will be affected by the evaluation. It should be 
remembered though, that not all decisions occur at the 
conclusion of the study. A fermentation period may be 
required prior to utilization. The essential concern is that 
the report reach the user prior to the decision period. 

The relevance issue can best be addressed by involving 
decision makers at the start of the evaluation. Emphasis 
should be on anchoring the evaluation to specific needs of 
the audience (s). Recognizing that there often are multiple 
audiences to the evaluation, the evaluation may have to focus 
on several different needs. Some may want to know about 
implementation fidelity, others about program 
accomplishments and still others about program costs. Each 
audience may have varied timelines and formats for reporting. 
Involving representatives of the various audiences in the 
design of the evaluation will ensure that it will be relevant 
to their needs. 

Communi c at i on • 

A related concept is the degree of communication between 
evaluators and clients. The utilization of evaluation 
results tends to increase when these persons engage in 
constant interchange about the evaluation. Such discourse 
should occur throughout the evaluation process from 
conceptualization and design through implementation and 
reporting. Information should come in several forms. 
Program administrators like to have results communicated 
verbally as well as in a formal report. 

Reports should be tailored to specific audiences. Program 
staff often need very specific data while program 
administrators respond better to overall statements of 
program achievement. Legislators tend to react more 
positively to executive summaries allowing their staff to 
read the technical reports. 
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Information Processing. 

There have been a number of special education articles in the 
past several years which claim that diagnostic data are not 
used in making educational decisions. No relationship can 
be found between the input data (test scores) and the output 
data (placement decisions). 

The reason for this finding may be found in the information 
processing literature. People receive information from 
diagnosticians and interpret the information given their 
professional experience and knowledge. This processing leads 
to a decision that is now based on new information, the 
processed information. 

The same may be true for evaluation. The users receive the 
report and must process the findings and recommendations in 
order to make judgments which lead to decisions. To 
facilitate processing, the evaluator needs to assist users 
in interpreting data. The user must be made aware of points 
of view which guided the evaluator 1 s thinking. 

Processing is enhanced when reports are specific to 
particular program elements, clearly presented and free of 
jargon. Information that is unexpected or unanticipated may 
have low utilization. Program administrators generally have 
some preconceived notions about how the evaluation will turn 
out based on their day to day association. If the findings 
are clearly different from expectations, then the user may 
be suspect. One way to counter this ocular trauma is to 
provide ongoing verbal feedback about the evaluation's 
findings. Reduce surprises! 

Finally, like it or not, some administrators tend to like 
qualitative in addition to quantitative descriptions. 
Utilization research refers to the extensive use of vivid 
examples in addition to quantitative findings. A proper 
response to this usage phenomenon is to design studies which 
provide both qualitative and quantitative data. 

Credibility. 

Can the user trust the evaluation results? What standards 
do the decision makers have for evaluating the evaluation? 
What evidence will be convincing? It was noted previously 
that the user is likely to have expectations regarding the 
outcome of the evaluation. In this sense the user may 
evaluate the results in terms of the extent to which the 
results confirm or extend expectations. 
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Combining the results of several evaluations can increase 
credibility. A state education agency that receives a number 
of reports indicating program success is more likely to 
decide in favor of continuing program support. Credibility 
in evaluation is increased by aggregated reports. 

Users who have a history of depending on evaluation findings 
appear more likely to continue using evaluation results. 
Success is a great motivator. Experiences in Virginia 
suggest that special education school staff who have positive 
experiences with evaluation want to continue. They report 
expanding their efforts to other aspects of their program as 
well as other areas of the system. 

The technical quality of the evaluation as a factor 
influencing use is interesting. Most researchers have found 
that technical quality is not a primary concern among users. 
The most often cited support for technical quality occurs in 
situations where a potential user must be convinced of the 
accuracy of the evaluation results. For example, a special 
education director may require convincing evidence of a 
program 1 s value prior to purchasing the product. Quality 
issues arise more often when summative judgments are being 
made as a result of evaluation. Finally , when people 
disagree with the findings of an evaluation, the attack 
typically centers on the quality of the methodology. 

User Involvement in Advocacy. 

To what extent are decision makers going to actively support 
the evaluation? How committed are decision makers to use? 
For example, in Virginia, a superintendent requested the 
evaluation of a particular program. After the evaluation got 
under way, the superintendent decided to retire. Interest 
in the evaluation dwindled. 

Evaluation results get used when someone, particularly a 
program administrator, champions the evaluation. When the 
evaluation findings are consistently reported in both formal 
and informal communication, utilization is enhanced. The 
same is true when groups of individuals within or outside the 
organization support the evaluation. 

Commitment is often related to the origin of the evaluation. 
If it is externally motivated, commitment is likely to be 
low. Take for example, Federal data reporting requirements. 
In one national study it was found that these data were used 
for state and local decision making. 

The way to handle this advocacy factor is to make sure you 
know the user prior to the start of the evaluation. The 
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evaluation and dissemination of results should be tailored 
to the needs of these users. Their involvement in the 
design, implementation and reporting of the evaluation will 
increase their perceived ownership of the efforts. They will 
have a stake in the use of the evaluation. They will know 
its strengths and weaknesses. They will become informed 
advocates of the evaluation. 
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STEPS TO PLANNING THE LOCAL EVALUATION. 



In this section keep practices which might be followed in 
planning the design / implementation and report of an 
evaluation of a local special education program are 
summarized. Planning for an evaluation can be summarized in 
eight steps as displayed in Figure 1 and briefly reviewed 
below. Worksheets to assist in planning local strategies to 
complete the steps are found in Appendix D and are referenced 
throughout the review. These steps are advisory in nature. 
They suggest that an evaluation is an orderly linear ~rocess. 
It's not* The user may employ some of the proposed sViAtegies 
and not others. At the same time strategies may be modified. 
Contextual factors associated with the evaluation guide its 
evolution. Information needs, time constraints, local 
resources and audiences are examples of factors which may 
influence the evaluation effort. 

The hope is that the reader's evaluation enterprise will 
follow the basic principles and practices discussed in 
previous sections. The assumption is that the user has 
identified a program which he or she needs to know something 
about. What appears on the following pages are some ideas on 
how to achieve this objective. It has been said that a 
drowning man will not pass up a patched life raft in hopes that 
an ocean liner will come by. What is presented here is a life 
raft. A complete 'Manual' on the evaluation of local special 
education programs in Virginia has been prepared by the author 
and is available from the Virginia Department of Education. 

One final note. Experience suggests that as the program 
evaluation unfolds changes are made in the program. That is, 
the process of asking questions as well as gathering 
information about a program is likely to lead to program 
modifications. This is OK! It is why people should enter into 
an evaluation effort. The desired product of an evaluation 
is program change based on information. At any point in the 
evaluation information may be generated which suggests some^ 
aspect if the program needs to be fixed. If so, fix it; it's 
not always necessary to wait until the full evaluation is 
completed to revise the program. 

1. Getting Started. 

The purpose of this stage is to get ready for the 
evaluation. The evaluation team must be selected. Who 
in the LEA will design, conduct and report the evaluation? 
Because of the complexities of the evaluation enterprise, 
the team should be composed of those persons familiar with 
the program as well as those trained in the practices and 
principles of evaluation. It is best if the team is 
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sanctioned by the school administration (see Worksheet 
II). 

The team must identify those persons who are interested 
in the program. These are the people who are or will be 
making judgments about the program. It is important to 
identify and specify the standards that will be used by 
these persons as they make judgments. Personal 
interviews, document reviews or surveys may be used. 

Once the team ir> selected, the initial focusing of the 
evaluation takes place. The team decides if the total 
program or certain of its elements/components are to be 
evaluated. Input from a variety of sources both internal 
and external to the program should be used. Previous 
evaluation studies, compliance monitoring reports are 
excellent beginning points (see Worksheet IV). 

The input from these multiple sources of information 
enables the team to establish the focus and purpose of the 
evaluation. The product of this 'getting started 1 phase 
of the evaluation is a list of the program component (s) 
which will be evaluated as well as a purpose statement 
which specifies who is interested in the evaluation of the 
component(s) and to what uses the evaluation information 
are directed (see Worksheet V). 

It should be noted that the focus of the evaluation may 
shift as the evaluation progresses. This is appropriate. 
However, when the focus changes, the purpose as well as 
interest groups will have to be reassessed. 

Describing the Program 

In this stage the team describes in detail the program or 
its components to be evaluated. The product is called a 
Program Design and serves as a map or blue print of the 
program that is followed by the team as they develop an 
understanding of the program. Again the Effectiveness 
Indicators document is an excellent resource for 
identifying particular components of the local program. 

The components of the program are described in terms of 
inputs (resources) , processes (activities), and outcomes 
(products or benefit.?, changes in programs or clients). 
Interrelationships between components are described. 
Inputs for some components may be outputs from others and 
vice versa (see Worksheet VI). 

Statements contained in the program description should be 
specific enough for anyone who reads them to understand 
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the program. The statements should include standards; 
expected / desired, or mandated program characteristics. 
These will serve the evaluation. The team will look' to 
these statements which come from regulations, guidelines, 
professional literature or professional experience for 
comparisons for future judgments about the program. 

After the design is completed, it should be reviewed by 
program staff and other interested parties for accuracy. 
It may be necessary to refocus the evaluation at this 
time. It also might be necessary to redefine the interest 
groups . 

3. Writing Evaluation Questions. 

Various types of evaluation questions might be addressed 
in evaluating a special education program. It should be 
remembered that evaluation is a comparative process; one 
which allows program performance to be compared to 
expectations regarding such performance. These 
expectations become the standards for the evaluation. The 
program description or design takes these standards into 
account and makes them explicit. Thus, the evaluation 
questions must be anchored in the program description if 
they are to yield useful information. 

Evaluation questions link the program design to the 
evaluation design. They serve as the vehicle through . 
which needed information is provided to the evaluation 
team. The evaluation questions focus the evaluation on 
specific elements of the program. They becoe the basis 
for the data collection strategies which are the core of 
the evaluation design. Thus, it can be seen that the 
evaluation questions are critical to the success of the 
evaluation. The central criterion for the success of an 
evaluation is its utility to decision makers. If the 
evaluation questions are inappropriately framed, then 
there is little probability that any useful information 
will result from the evaluation. 

What kinds of questions can be asked in the evaluation of 
special education programs? Many people wait until the 
program is in operation or has been completed to conduct 
an evaluation. In the main, their focus is on outcome; 
"Are we achieving our objectives?". In our opinion it is 
unnecessary, if not foolish, to focus only on outcome 
evaluation questions. There are many types of evaluation 
questions which might be addressed in the evaluation 
depending on the information needs of decision makers. 
An excellent source for determining the types of 
evaluation questions which might be addressed in the 
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evaluation of a local special education program is the RRC 
Effectiveness Indicators document referred to earlier. 



3.1 Design Evaluation Questions. 



The first type of evaluation concern arises at the program 
design stage of program development* The purpose here is 
to judge the quality of the program prior to its 
implementation. This kind of information can save program 
staff from going down blind alleys and therefore, conserve 
their time and their client 1 s time. There are four basic 
questions that can be asked at the design stage: 

a) Is the program design accurate ? Information needed 
here centers on the degree to which program staff, 
administrators, and others interested in the program 
agree with the description of the program 1 s inputs, 
processes and outputs as well as the 

/ interrelationships which exist between program 

elements or components. The goal is consensus among 
administrators and others as to overall accuracy of 
the description. 

b) Is the program technically sound ? In this question 
the focus is on the theoretical basis for the program. 
Given what we know as best practice, either from 
experience or research, will this program work? When 
Experts 1 view the program do they find it technically 
sound? 



c) Is the program design complete and internally 

consistent ? The intent of this question is two fold. 
The first to determine the extent to which the 
description of program components contains all 
necessary elements to enable the program to meet 
relevant standards. Does the program include 
references to state, federal and/or professional 
standards for providing services to exceptional 
learners. 

The second focus of this question is on the logical 
consistency of the program. Essentially, it addresses 
the relationships between and among program functions. 
If the outcome of one component becomes the input to 
another, is the dependency accounted for in the 
program description? For example, is there a logical 
relationship between resource room and regular 
classroom experiences for learning disabled students? 



d) Is the program politically sound ? Here we are 

interested in fittedness. That is, to what degree 
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does the program description take into account the 
needs and expectations of boundary programs. The 
special education program exists within a broader 
context. It must be framed so that it can coexist 
with other programs. Relationships must be such that 
special education can benefit from the resources of 
other programs and at the same time be a benefit to 
those programs. This fourth design evaluation 
question seeks to gather information concerning any 
barriers to such co existence. For example , do 
special education training experiences mesh with 
regular programs? 

If these design evaluation questions are addressed, then 
the program will have greater probability for success upon 
implementation. It should be noted/ however, that these 
questions may also be asked of programs in the operational 
phase of their development. Answers to these questions 
can identify the causes for real or potential problems at 
any programming phase. 

.2 Implementation Evaluation Questions. 

As the program is implemented, it is important to 
ascertain two things. First, the program staff must 
determine if all required resources or inputs are present. 
The program cannot be operated without these inputs. The 
required inputs are found in the program description. The 
following represent types of questions that might be 
asked: 

• Do special education students have the skills 
required to enter the program? 

• Are there the appropriate number of staff with 
required competence? 

• Are required facilities available? 

• Did the program staff acquire necessary 
instructional materials? 



• Are there sufficient funds to operate the program? 

Second, the program staff must determine the degree to 
which inputs are allocated according to the expectations 
(standards) established in the program plan or design. 
The form as well as the schedule of the program becomes 
the focus of this aspect of the evaluation effort. Are 
program activiti es beina implemented according to the 
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program design, e.g., are students being mainstreamed in 
the manner described by staff? 

.3 Outcome Evaluation Questions. 

Now the focus shifts to program results or effects, both 
interim or ongoing and final . There are seven basic 
questions which might be addressed in this phase of the 
evaluation: 

a) Are we achieving goals as predicted ? Here the 
evaluator is centering on interim or enroute 
objectives. The program is in its operational phase. 
This evaluation may occur at any time during the 
program. The program description will include 
anticipated performance outcomes with relevant 
timelines. These questions are particularly important 
if future performance depends on the acquisition of 
prior skills, knowledge, attitudes, behaviors or other 
changes in the program 1 s targets. Implementation 
evaluation questions help interpret the findings 
related to these process evaluation questions. The 
focus here might be on short term IEP goals. 

b) Did we achieve our goals ? The difference between this 
question and the previous one is that the question is 
asked at program completion and addresses terminal 
objectives. In most education programs the terminal 
objectives are arbitrarily set. Thus, we might be 
focusing on annual goals as stated in the learner's 
IEP. 

c) Did we cause the changes in the target? Here the 
emphasis is on proving a causal relationship between 
the program's intervention techniques and the* 
outcomes. The evaluator' s task is to rule out any 
rival or competing hypothesis or explanation. That 
is, the evaluator must answer the question, "Are there 
any reasonable explanations other than the program 
intervention strategies regarding the cause for 
changes in the program's targets?". 

d) How is the program perceived ? The focus here is on 
people's opinion of the program. Attitude assessment 
of program participants and others is conducted. The 
evaluator is interested in how people are reacting 
to the program. Do trainees perceive the program as 
being beneficial to their professional development? 
Do administrators believe the program is valuable? 

Do parents view the program as beneficial for their 
exceptional learner? 

18 
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e) What did the program cost ? This question seeks to 
describe the program costs. Typically, the focus is 
on start up costs, those one time costs that are 
necessary to establish the program, and operational 
costs, those that are recurring throughout the program 
such as staff salaries or consumable supplies. 

f ) Is the program cost effective ? Answering this 
question is very difficult* A number of things must 
fall into place before the question can be answered. 
The program is compared to another program with 
identical objectives. If the program results in equal 
or better achievement of the objectives at a lower 
cost then it is said to be cost effective. 

The problem with this approach is two fold. First, 
the evaluator has to find a comparison program with 
identical objectives. Second, it must be established 
that each program causes the projected benefits. An 
additional concern beyond cost, is the side effects 
of the programs being reviewed. A program may be less 
costly, but may give rise to negative staff or 
consumer reactions which would inhibit the 
implementation and effectiveness of the program in a 
new site. 

g) Are trainees successful after leaving the program? . 
These follow up evaluation questions focus on program 
graduates. Although they may focus on learners who 
leave the special education program to re enter 
regular education or leave school and enter into post 
school experiences such as the world of work or higher 
education, they may also look at the impact inservice 
education programs have on classroom instruction. For 
many decision makers, particularly those outside of 
the program, answers to these questions represent the 
'proof of the pudding 1 . 

As can be seen, there are a number of different questions 
which might be addressed in the evaluation of an 
educational program. The 'need to know 1 function of the 
evaluation dictates which questions are asked. That is, 
there are many people who may need to know something about 
the program. These people were identified in the previous 
section. It is important to involve these people in the 
identification and the formation of the evaluation 
questions. These questions focus the evaluation design. 
If people are omitted at this stage, then their 
information requirements may not be met (see Worksheet 
VII) . 
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4. Planning Collection of Information . 



The central focus here is matching the information/data 
collection to the information need expressed throughout 
the evaluation questions. Remember that the questions are 
linked to program elements. If the match occurs then the 
evaluation is more likely to yield usable results. 

There are many different, complementary types of data 
collection strategies, both formal and informal. These 
include interviews, tests, surveys, observations and 
record reviews. 

The team must plan to collect information that is 
convincing/believable to the evaluation audience. 
Concepts of representativeness, reliability, validity and 
objectivity must be considered when developing or 
selecting the data collection strategies. The team must 
plan to field test all data collection strategies prior 
to their actual use in the evaluation. Additionally, the 
team must plan to collect information in a timely fashion. 
That is, it must be gathered and presented within a 
timetable which will facilitate utilizing it for decision 
making, the purpose of evaluation (see Worksheet VIII). 

5. Planning the Anal ysis of Evaluation Data. 

The evaluation will result in the collection of a 
considerable amount of data from various sources. The 
purpose of data analysis procedures is to reduce raw data 
to a manageable form to allow for interpretations and/or 
inference with regard to the evaluation questions. A 
number of statistical techniques and data presentation 
methods are available to the team. These should be 
considered prior to actual data collection. Indeed, it 
is useful to field test the collection practices to 
determine if anticipated analysis techniques will be 
appropriate. Also, the technique and presentation 
strategies should be shared with audience representatives 
to determine their utility for judgment and decision 
making tasks. 

An analysis plan should be developed by the team. The 
plan should be limited to each individual data collection 
process (instrument, interview, observation, etc. ) . It 
should identify potential interpretations and analyses to 
be made, person(s) responsible for the analyses, dates by 
which the analyses must be completed, and persons who must 
receive them (see Worksheet IX). 
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Planning the Evaluation ReoortfsK 



Reporting is the final step in the evaluation process. 
When planning a report, the team has two objectives: to 
describe the methods and findings of the evaluation in 
relation to the questions posed in the evaluation; and, 
to recommend actions which might be taken to overcome any 
discrepancies identified in the evaluation. 

Reporting closes the evaluation cycle. That is, it is the 
vehicle to get information to evaluation audiences. 
Therefore plans must be made to ensure that it is 
readable, comprehensible and timely. If not, the 
evaluation will fail its purpose to gather information to 
assist decision makers. 

As noted earlier, the team may plan reports that are 
formal or informal and may plan to use varied formats 
including written, verbal and/or audio-visual 
presentations. Agreements regarding report ^ 
characteristics should be made prior to the implementation 
of the evaluation. Additionally, the audience(s) for the 
report should be identified at the start. Any agreements 
associated with respondent identification or anonymity 
should be made prior to the study and maintained 
throughout, including the report (see worksheet X). 

One final comment regarding the report. The evaluator'.s 
responsibility extends beyond the reporting •process 1 . 
It is important for the evaluator to work with the 
receptors of the report to assist them in interpreting the 
processes and products of the evaluation in terms of the 
objectives of the evaluation as well as the ongoing and 
continuing program decisions and evaluation needs of the 
program administration staff and other parties interested 
in the program. Most programs with which are evaluated 
are like rivers which keep flowing. Our evaluation is but 
a ripple in its surface. The next look may involve a 
totally new context in which to work. 

Managing the Evaluation. 

As can be seen, the evaluation enterprise is complex. In 
order to maintain some sense of sanity and rise above the 
confusion, it is necessary to develop some type of 
management plan. There are two primary components to the 
management plan: a schedule of evaluation events and a 
budget. The plan will help organize the team's efforts. 
When this plan is completed, the team and others 
interested in the evaluation will be able to use it to 
assess logical consistency between and among tasks as well 
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as the reasonableness of timelines and costs before the 
evaluation begins. The plan will serve as a monitoring 
guide as the evaluation progresses. 

The schedule of events is tied to the evaluation 
objectives or questions. For each there are sub- tasks or 
sub-questions. For each of the sub-tasks, the team 
identifies the person(s) responsible for the activity and 
its expected start and finish dates. This information can 
be used as the standard for the meta evaluation of the 
effort which is described at the end of this document. 
The schedule should include expected objectives and times 
for meta evaluation. (see Worksheet XI). 

The budget for the evaluation contains the same elements 
as most programs. Generally there are five components 
(see Worksheet XII): 

• Salaries & Benefits 

• Travel & Subsistence 

• Materials <£ Supplies 

• Other expenses including: 

— telephone 

— postage 

— copying/printing 

— computer 

— honoraria for respondents/consultants 

• Indirect Costs 
Meta Evaluation . 

The purpose of evaluation is to gather information upon which 
judgments can be made regarding the worth of an object 
(educational program, product, or process). When the 
evaluation process was discussed, it was indicated that it 
results in judgments which are comparative in nature* 
Standards or statements of expectation are compared to 
performance or what 'is 1 . 

The same can be said of meta evaluation. In this case the 
judgment is a comparison made between the 'real 1 evaluation 
effort and expectations of what the evaluation should be. 
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There are standards for good evaluations just like there are 
standards for good programs. The Joint Committee on Standards 
for Educational Evaluation has developed standards for judging 
the merit of evaluation enterprises; utility, feasibility, 
accuracy and propriety. 4 These standards should be employed 
as guides in the development of the Evaluation Design. Other 
standards for special education programs come from federal and 
state regulations, professional standards, private 
accreditation services, best practices manuals and the 
professional literature* These standards should be 
considered when the Evaluation Design is constructed* The 
Evaluation Design becomes the standard for the meta 
evaluation* 

Perhaps the most important standard for the meta evaluation 
comes from the decision maker, the audience for the 
evaluation* The standard here is usefulness* Does the 
evaluation produce the information which is required by the 
decision makers? The local evaluation team could develop a 
very elegant evaluation design but in the end the decision 
makers have to believe that the design will lead to answers 
to their evaluation questions. The evidence or information 
must be convincing to them. This is the primary standard to 
use in judging the worthiness of an evaluation enterprise. 

We indicated earlier that the stages of evaluation follow the 
stages of program development, implementation and completion. 
The same is true for meta evaluation efforts* In fact, the 
basic steps in meta evaluation occur when the evaluation plan 
is written, before it is installed (design evaluation), when 
it is in operation (process evaluation) and when it is 
completed (outcome evaluation). The same types of questions 
can be asked in a meta evaluation as those addressed in 
section F-l. The reader should review these questions in 
light of the purpose of meta evaluation (see Worksheet XIII). 

The initial meta-evaluation question often posed by the local 
evaluation team is "What are the driving and restraining 
forces to conducting the planned evaluation?" After 
completing the Management Plan the team will know precisely 
what is needed to conduct the evaluation. Now they must 
determine what factors or forces exist to support the 
evaluation and those which impede or restrain the successful 
implementation and use of the evaluation* It is important to 



Joint Committee on Standards for Educational Evaluation. 

Standards for Eva luations of Educational Programs. Projects 
and Materials . New York: McGraw Hill 1981. 
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involve stakeholders in this Force Fi eld Analysis (see 
Worksheet XIV). 



The meta evaluation can be both informal and formal. 
Sometimes it is useful to informally discuss the evaluation 
plan or progress within the team or with representatives of 
target audiences. At other times it may be necessary to 
employ a third party evaluator, one who is neutral to both the 
program and the evaluation team. Formal and informal 
reporting mechanisms can be used. Just as programs are 
monitored as they are implemented / an evaluation can be 
monitored with reports coming at anticipated checkpoints. 

For example , when a school system develops a special project 
to the point they believe it is good enough to share with 
others , they may call in an external evaluation team to audit 
the results of the school's evaluation efforts. This will 
facilitate dissemination activities. It may enhance the 
credibility of the claims made by the school regarding program 
effectiveness. 

Evaluation planning requires many activities: focusing, 
describing, developing questions, data gathering strategies 
and analysis techniques. All of these should become the focus 
of meta evaluation activities. For example, are the 
evaluation questions clear enough to allow formation of data 
gathering strategies? Do decision makers agree that proposed 
strategies will meet their needs? Are the timelines for 
gathering data appropriate? Is the cost of the evaluation 
appropriate to the cost of the program? 

A checklist developed by Blaine R. Worthen for conducting meta 
evaluation is set forth in Appendix A. The checklist could 
be used by program staff or audiences of the evaluation to 
determine its worth. 
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Appendix A 



CHARACTERISTICS OF GOOD EVALUATION STUDIES 5 
Blaine R. Wo r then 
Northwest Regional Education Laboratory 



YES 



NO 



1. Conceptual Clarity —Ref era to whether or not 
the evaluator exhibits a clear understanding 
of the particular evaluation being proposed. 

2. Characterization of the Object of the 

^valuation — No evaluation is complete unless 
it includes a thorough, detailed description 
of the program or phenomenon being evaluated. 

3. Recogniti on and Representation of Legitimate 
Audiences — An evaluation is adequate only if 
it includes input from and reporting to all 
legitimate audiences for the evaluation. An 
evaluation of a school program which answers 
only the questions of the school staff and 
ignores questions of parents, children, and 
community groups is simply a bad evaluation. 

4. Sensitivity to Political Problems in 
Evaluation — Many a good evaluation, 
unimpeachable in all technical details, has 
failed because of its political naivete. 

5. Specification of Information Ne eds and 
Sources — Good evaluators tend to develop and 
follow blueprint which tells them precisely 
what information they need to collect and what 
the sources of that information are. 

6. Comprehensiveness /Inclusiveness— The wider the 
range and the more important the variables 
included in the evaluation, the better it 
generally is. 

7. Technical Adequacy — Good evaluations are 
dependent on construction or selection of 
adequate instruments, the development of 
adequate sampling plans, and the correct choice 
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Dr. Worthen is now at Utah State University, Department of 
Psychology. 
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and application of techniques for data 
reduction and analysis* 

8. Consideration of Costs — Educators are often 

faulted for choosing the most expensive program 
from two that are equally effective, just 
because the expensive one is packaged more 
attractively or has been more widely 
advertised. The real fault lies with the 
evaluations of those programs which fail to 
consider cost factors along with the other 
variables. 

9. Explicit Standardg/Criteria»"A statement of the 

criteria or standards which are used to 
determine whether the program was a success or 
a failure. The measurements and observations 
taken in an evaluation cannot be translated 
into judgments of worth without the application 
of standards or criteria. 

10. Judgments and/or Recommendations— The only 

reason for insisting on explicit standards or 
criteria is that they are the stuff of which 
judgments and recommendations are made, and the 
latter are the sine qua non of evaluation. 

11. Reports Tailored to Audiences— A typical 

evaluation might end up with one omnibus 
technical evaluation report which 
self-consciously includes all the details and 
one or more non-technical evaluation reports 
aimed at the important audience(s). 
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Utilization of Evaluation 

Partial Reference List 

Ciarlo, James A. (ed. ) Utilizing Evaluation: Concepts and 
Measurement Techniques. Beverly Hills, CA: Sage, 1981. 

Cousins, J. B. and Leithwood, K. A* Current Empirical 
Research on Evaluation Utilization. Review of Educational 
Research, Fall 1986, Vol. 56, #3, pp. 331-364. 

Cuba, E. G. and Lincoln, Y. S. Effective Evaluation: 
Improving Usefulness of Evaluation Results Through Responsive 
and Naturalistic Approaches. San Francisco, CA: Josey-Bass, 
1981. 

Joint Committee on Standards for Educational Evaluation. 
Standards for Evaluations of Educational Programs, Projects 
and Materials. New York: McGraw-Hill, 1981. 

Levi ton, L. C. and Hughes, E. F. Research on the Utilization 
of Evaluations: A Review and Synthesis. Evaluation Review, 
Vol. 5, No. 4, August 1981. 

Patton, M. Q. Utilization Focused Evaluation (2nd Edition). 
Beverly Hills, CA: Sage, 1986. 

Rossi, P. H. and Freeman, H. E. Evaluation: A Systematic 
Approach (3rd Edition). Beverly Hills, CA: Sage, 1986. 

Weiss, C. H. Measuring the Use of Evaluations, pp. 129*146 
in E. House (ed.) Evaluation Studies Review Annual (Vol. 7). 
Beverly Hills, CA: Sage, 1982, 



27 



Appendix C 

Strategies to Enhance Utilization 

The following strategies have been found to affect the 
utilization of evaluation studies. Much of the information 
is adapted from a treatment of the topic by Peter Rossi and 
Howard Freeman in their text entitled "Evaluation: A 
Systematic Approach: Sage Publications, 1985. It should be 
remembered that evaluations serve many purposes and their use 
indices may either fall into the instrumental or conceptual 
domains. Instrumental use suggests that direct impact of the 
evaluation can be documented in some specific beha\ioral, 
organizational or system change. Conceptual uses center more 
on influencing the receptor's thinking about the entity that 
is being evaluated. According to Michael Patton, these uses 
are best described as 'reducing uncertainty 1 in the decision 
maker regarding a program. Finally, utilization is not time 
bound. The evaluation may affect the program before the 
evaluation is implemented, while it is being implemented, 
immediately after the evaluation, and/or sometime after its 
conduct. Thus, measurement of utilization will necessarily 
have to be continuous with utilization tracers* constantly 
available to monitor usage. 

1.1 The evaluators should identify and involve decision 
makers at the start of the evaluation process. 

While some have suggested that decision makers may not 
want to be involved in the evaluation, most would agree 
that utilization is enhanced by identifying the 1 users 1 
of evaluation at the start. They should be allowed to 
become involved in the: 

a. identification of specific attributes of the program 
to be addressed in the evaluation; 

b. generation of evaluation questions; 

c. setting of standards for each question; 

d. establishment of data gathering and analysis 
procedures ; and, 

e. selection of report time frames and formats 
appropriate to their needs. 

1.2 Evaluation designs must be feasible. 

The scope of the evaluation must match the program to be 
evaluated. The requirements (costs, staff and 
participant time) should not be unduly burdensome. The 
costs should be reasonable given the size and importance 
of the program. Disruptions in normal routine should be 
minimal. The concerns and values of those affected by 



28 



the evaluation should be assessed at the start and be 
reflected in the evaluation. 

1.3 The evaluation should be conducted in a legally and 
ethically responsible way. 

Propriety is the concern here. Protection of the rights 
of those participating in the evaluation is emphasized. 
Persons involved in the evaluation as well as those who 
are audiences for the evaluation have a right to know 
about the evaluation throughout its life cycle. 

1.4 Evaluations should be technically sound. 

The data gathering procedures should be reliable, valid 
and objective. They should logically flow from the 
evaluation questions addressed. Discussions, 
conclusions and recommendations must be logically related 
to results and evaluation questions. 

1.5 Utilization and dissemination plans should be an integral 
part of the evaluation from the start. 

Uses of the evaluation should be anticipated by program 
evaluators and decision makers from the beginning. 
Tracers which monitor and document both anticipated and 
unanticipated outcomes of the evaluation should become 
part of the evaluation design. Agreements concerning who 
receives evaluation reports, when and in what format 
should be made prior to implementing the evaluation. 
Those agreements should be adhered to throughout the 
evaluation. Reports must be available when data are 
needed for decision making and should be tailored to the 
expectations of the audience. Reports should be jargon 
free and include an executive summary. 

1.6 The evaluation staff should be cognizant of contextual 
factors which might facilitate or impede information 
usage. 

Often the usage of evaluation information is affected by 
factors outside of the control of ths evaluators. The 
organization may not be ready for immediate change. 
Staff organization may not be ready for immediate change. 
Staff turnover may result in different priorities. Short 
falls in expected revenues may reduce opportunities for 
change. Public pressure may push for change even though 
it is not called for. Thus, decisions about the program 
may not follow the recommendations of the evaluations or 
may concur, but for different reasons. 
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1.7 The evaluation staff should view themselves as change 
agents . 

Often the reason for reduced impact of the evaluation is 
that the evaluators do not- assist program staff in 
understanding and interpreting the evaluation findings 
in terms of program decisions. This often occurs with 
terms of program decisions. This often occurs with 
evaluators who have 1 hired gun 1 status. These evaluators 
are usually on to another program even before they finish 
the current evaluation. They do not stay around long 
enough to assist in the change process. Thus, it becomes 
the responsibility of program staff or the sponsor of the 
evaluation to articulate the findings of the evaluation 
with program improvement decisions. 

1.8 The perceptions of the user will influence the degree to 
which evaluation results impact on programs. 

The evaluation staff must have credibility. This may be 
established both through prior history and the design of 
a functional evaluation design. Continuous interaction 
with decision makers throughout the evaluation will 
increase credibility. 

Credibility is enhanced by having an interdisciplinary 
team responsible for the evaluation. This team might 
include program professionals and evaluation 
specialists. Further, the team's efforts should be 
sanctioned by both program administrators and higher 
level administrators such as local school boards or 
finding agency representatives. 

The decision maker's previous experience with evaluation 
may drive or restrain the use of the evaluation. The 
evaluator may have to reverse negative 

expectations/skepticism on the part of all or some users. 
At a minimum, the evaluator will have to identify 
perceptions of f good evaluation 1 within the user group. 
If these perceptions vary to great extent from the 
evaluator, then this may be reason to discontinue the 
evaluation. 

1.9 Multiple source/multiple methods broaden the 
acceptability of evaluations. 

For each evaluation there are likely to be several 
audiences which represent people who will make judgments 
about the program. The evaluation design serves to 
generate data upon which these judgments will be made. 
When different data gathering strategies (e.g., 
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interview, survey, and/or record review) and different 
sources (e.g., parents, teachers and administrators) are 
employed to answer the same evaluation question the 
information generated becomes more credible and may meet 
the needs of varied audiences, thus enhancing use of the 
evaluation. 

10 The evaluation should be anchored to specific program 
elements* 

So often we encounter program administrators and staff 
who say, "Now that we have the data, what should we do 
with them?" This generally occurs when the request for 
the evaluation is external to the program. Thus, someone 
outside the program funding agency representatives, 
legislators, or community members wants to know about the 
program. However, it happens with equal frequency when 
the evaluation questions are not linked to particular 
program components prior to the start of the evaluation. 
As indicated previously, the decisions to be served by 
the evaluation should be identified prior to the 
implementation of the evaluation by both evaluators and 
decision makers. These decisions should be anchored to 
specific elements of the program. Are decisions going 
to be made regarding program inputs (staff, clients, 
funds, equipment, other program resources), processes 
(strategic or activities) and/or outcomes (anticipated 
changes in clients, organization or systems)? To 
understand and utilize the evaluation in terms of these 
elements it will be necessary to explain each in enough 
detail to allow the explanation to become a standard for 
the evaluation. 

.11 Information retrieval and processing systems should be 
reviewed. 

Several studies cite inadequate or inappropriate 
information systems within the user agency to facilitate 
use of evaluation results. This may be true for all 
levels of evaluation, but may be particularly important 
for system wide studies. For example, when a state 
special education department contemplates a state-wide 
study, it should assess its information processing 
capabilities. Will it be able to receive, process and 
disseminate information generated from the evaluation. 
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